Atlas search lookups #325

WaVEV · 2025-06-24T14:29:51Z

This PR adds the initial implementation of the Atlas operator.

Task:

django_mongodb_backend/functions.py

django_mongodb_backend/compiler.py

tests/queries_/test_search.py

django_mongodb_backend/expressions/builtins.py

docs/source/ref/models/search.rst

timgraham · 2025-08-05T15:00:55Z

docs/source/ref/models/search.rst

+``SearchEquals`` objects can be reused and combined with other search
+expressions.
+
+See :ref:`search-operations-combinable`


I wonder if we could structure things so we don't need to repeat this boilerplate on every(?) expression.

🤔 I think we cannot scape, unless we list the operations that could be combined in the section of combinable operations. I like to have this link meanwhile I am reading the docs, so it gives an introduction of some (cool?) behaviour

docs/source/ref/models/search.rst

tests/queries_/test_search.py

timgraham · 2025-08-05T15:10:37Z

tests/queries_/test_search.py

+    delayedAssertCountEqual = _delayed_assertion(timeout=2)(TransactionTestCase.assertCountEqual)
+    delayedAssertListEqual = _delayed_assertion(timeout=2)(TransactionTestCase.assertListEqual)
+    delayedAssertQuerySetEqual = _delayed_assertion(timeout=2)(
+        TransactionTestCase.assertQuerySetEqual
+    )


Are the non-delayed versions ever used? Maybe it's better to overwrite the original names so we don't have to write "delayedXXXXX" everywhere. Or maybe the waiting could be done in setUp() after data is inserted? Unless some test inserts more data, essentially only the first test's waiting is needed, right?

No, I all the checks are delayed...
Regarding to the second question: right, any test that insert data need to wait. If the data is inserted in the init class, we could only wait once. So If we want to get rid of those delayed, we can wait in the creation part.

timgraham · 2025-08-05T15:12:06Z

tests/queries_/test_search.py

+
+
+@skipUnlessDBFeature("supports_atlas_search")
+class SearchEqualsTest(SearchUtilsMixin):


I've tried to be consistent in this project about using "Tests" (plural) in the class names.

🤔 mmh I didn't notice that. will change.

timgraham · 2025-08-05T15:16:37Z

tests/queries_/test_search.py

+        boost_score = SearchScoreOption({"boost": {"value": 3}})
+
+        qs = Article.objects.annotate(
+            score=SearchEquals(path="headline", value="cross", score=boost_score)
+        )


I'd inline boost_score, or at least omit the blank line. (Only some tests are inconsistent.)

Jibola

Things look great, but I've gone through about half of the code (due to size). I will check the test code tomorrow!

django_mongodb_backend/expressions/builtins.py

django_mongodb_backend/compiler.py

Jibola · 2025-08-05T18:32:33Z

django_mongodb_backend/compiler.py

+            if not has_search:
+                raise ValueError(
+                    "Cannot combine two `$vectorSearch` operator. "
+                    "If you need to combine them, consider restructuring your query logic or "
+                    "running them as separate queries."
+                )
+            raise ValueError(
+                "Only one $search operation is allowed per query. "
+                f"Received {len(search_replacements)} search expressions. "
+                "To combine multiple search expressions, use either a CompoundExpression for "
+                "fine-grained control or CombinedSearchExpression for simple logical combinations."
+            )


I think these two ValueErrors need to be switched.

🤔 the second is the case when:
has_vector_search but it does not has search. I think I should refactor this. It is a bit confusing. the not at the beginning is not helping.

django_mongodb_backend/compiler.py

django_mongodb_backend/expressions/builtins.py

django_mongodb_backend/compiler.py

django_mongodb_backend/expressions/search.py

Jibola · 2025-08-05T21:28:53Z

django_mongodb_backend/expressions/search.py

+        # Apply De Morgan's Laws.
+        operator = node.operator.negate() if negated else node.operator
+        negated = negated != (node.operator == Operator.NOT)


This logic is a little confusing because it requires some understanding of negate and the state changes.
I'll leave this as a comment here to be reviewed later.

What's an example of a NOT combinable?
I.e., how would I construct NOT (A AND B) or can this only be done via negate?

I applied De Morgan's Law to get something in the scope of A' operator B'. So:
NOT (A AND B) = Not A or Not B => {SHOULD: [MUST_NOT(A), MUST_NOT(B)] with minimum should in 1.
The other way to handle this is push everything in a must not, but in order to handle: NOT (NOT A)) as A I decided to apply this kind of simplifications.

And, I almost forgot, we could handle double negation on should. (That are the ors if the minimumShouldMatch is 1 ). long story short
A and B => MUST
not C => MUST_NOT
A or B => SHOULD with minimumShouldMatch is 1
not (A or B) => not A and not B => MUST(MUST_NOT(A), MUST_NOT(B))

When A, B, C are atomic.

I refactored it a bit. It is simpler now, don't know if it is simple enough 😄

django_mongodb_backend/expressions/search.py

Jibola

Overall PR looks great! I've got some minor corrections, but other than that, it is good to merge from me. Great work! 🚀

It also looks like there's a ReadTheDocs error:

/home/docs/checkouts/readthedocs.org/user_builds/django-mongodb-backend/checkouts/325/docs/source/ref/models/search.rst:654: WARNING: unknown document: 'atlas:atlas-search/scoring/' [ref.doc]

Jibola · 2025-08-06T14:54:00Z

docs/source/releases/5.2.x.rst

@@ -16,6 +16,12 @@ New features
 - Added :class:`~.fields.PolymorphicEmbeddedModelField` and
  :class:`~.fields.PolymorphicEmbeddedModelArrayField` for storing a model
  instance or list of model instances that may be of more than one model class.
+- Added support for MongoDB Atlas Search expressions, including
+  ``SearchAutocomplete``, :class:`.SearchEquals`, ``SearchVector``, and others.


Suggested change

``SearchAutocomplete``, :class:`.SearchEquals`, ``SearchVector``, and others.

``SearchAutocomplete``, :class:`SearchEquals`, ``SearchVector``, and others.

This suggestion isn't correct. Without the leading dot, the class won't be resolved properly.

🤔 maybe I forgot to add the prefix ~.expressions. But I don't know much about docs.

It works as is. The dot allows the path to be looked up rather than resolved as an exact match.

Jibola · 2025-08-06T15:18:21Z

tests/queries_/test_search.py

+    def create_search_index(cls, model, index_name, definition, type="search"):
+        collection = cls._get_collection(model)
+        idx = SearchIndexModel(definition=definition, name=index_name, type=type)
+        collection.create_search_index(idx)


NIT: For the sake of testing, we can make this a blocking call and check for the index before continuing.

🤔 If I don't understand wrong, I have to add a wait for predicate. Right?

tests/queries_/models.py

tests/queries_/test_search.py

timgraham reviewed Jun 25, 2025

View reviewed changes

django_mongodb_backend/functions.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 25, 2025

View reviewed changes

django_mongodb_backend/functions.py Outdated Show resolved Hide resolved

WaVEV force-pushed the atlas-search-lookups branch from 449b6a3 to ca8a7cf Compare June 26, 2025 02:56

WaVEV commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/compiler.py Show resolved Hide resolved

WaVEV force-pushed the atlas-search-lookups branch 3 times, most recently from 9935b25 to a467a57 Compare July 12, 2025 23:32

WaVEV changed the title ~~[WIP] Atlas search lookups~~ Atlas search lookups Jul 14, 2025

WaVEV force-pushed the atlas-search-lookups branch 4 times, most recently from ea2118b to 206b554 Compare July 21, 2025 19:29

timgraham reviewed Jul 22, 2025

View reviewed changes

WaVEV force-pushed the atlas-search-lookups branch 4 times, most recently from 456028d to 65f22e6 Compare July 22, 2025 05:16

WaVEV marked this pull request as ready for review July 24, 2025 19:39

WaVEV force-pushed the atlas-search-lookups branch from eb6eb07 to e7f4d22 Compare July 26, 2025 02:40

WaVEV force-pushed the atlas-search-lookups branch from 0fdb066 to eed2499 Compare August 5, 2025 00:25

timgraham and others added 10 commits August 5, 2025 10:34

Create django_mongodb_backend.expressions package

b774f21

Adapt query and compiler for operator support.

007ad74

Add Search operators.

2b78216

Make operators combinable and add compound expressions.

6211d89

Add vector search operator.

1828ee4

Add search lookup.

d4c2743

Add test search

23f90dd

Add combinable test

67f0bf3

Test clean up.

fd791f3

Add dalayed assertion methods in unit test.

a7af873

WaVEV added 8 commits August 5, 2025 10:34

Support operator as string

67f9c86

Update docstring

6b99ecf

Add docs

11cc8bf

Edit docs.

6d8edba

Update docs.

366c151

add available_apps to unit test

a937cb1

Simplify clean up call

c195010

Add change log

99f6548

WaVEV force-pushed the atlas-search-lookups branch from eed2499 to 99f6548 Compare August 5, 2025 13:35

doc edits

2b8e2b0

timgraham reviewed Aug 5, 2025

View reviewed changes

Jibola reviewed Aug 5, 2025

View reviewed changes

WaVEV added 2 commits August 6, 2025 09:27

Fixed Docs

2baafcf

Docstring and minor changes.

b491d6e

Jibola requested changes Aug 6, 2025

View reviewed changes

WaVEV added 9 commits August 7, 2025 11:21

Updated logs.

d06ca78

Refactor unit test.

c035a6c

Rename the delayed methods.

eac06d1

Fix broker references.

ed741f5

Fix docs lines

d869ed0

Move search test to atlas_search_ test app

a08d556

Docstring

be5c4b7

Fix docstring

61888b5

Simplify logic

0dc6ace



		@skipUnlessDBFeature("supports_atlas_search")
		class SearchEqualsTest(SearchUtilsMixin):

	``SearchAutocomplete``, :class:`.SearchEquals`, ``SearchVector``, and others.
	``SearchAutocomplete``, :class:`SearchEquals`, ``SearchVector``, and others.

Atlas search lookups #325

Are you sure you want to change the base?

Atlas search lookups #325

Conversation

WaVEV commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jibola left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jibola left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jibola Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WaVEV commented Jun 24, 2025 •

edited

Loading

Jibola left a comment •

edited

Loading

Jibola Aug 6, 2025 •

edited

Loading